DOCUMENT RESUME 

ED 251 504 * TM 840 801 



AUTHOR 
TITLE 

INSTITUTION 

SPONS AGENCY 
REPORT NO 
PUB DATE 
CONTRACT 
NOTE 

PUB TYPE 

EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



Milazzo, Patricia; And Others 

Methodology for Analysis of lAI District Level Data 
Bases. 

Southwest Regional Laboratory for Educational 
Research and Development, Los Alamitos, Calif. 
National Inst, of Education (ED), Washington, DC. 
.SWRL-TR-70 
6 Apr 81 
NEC-00-3~0064 
23p. 

Reports - Research/Technical (143) 
MFOl/PCOl Plus Postage. 

*Academic Achievement; Administrative Policy; Data 
Analysis; *Databases; Elementary Second ry Education; 
♦Evaluation Methods; * Instruct ional Development; 
Racial Composition; Racial Differences; *School 
Districts; Test Results 

♦Instructional Accomplishments Information Systems; 
*Los Angeles Unified School District CA; Survey of 
Essential Skills 



ABSTRACT 

Instructional Accomplishment Information (lAl) 
Systems data bases provide the opportunity for new and powerful 
studies relevant to educational policy issues at a local and/or 
national level. This report discusses the methodology for "schooling 
policy studies." The procedures are illustrated using a 
yet-to-be-completed analysis of the Los Angelas Unified School 
District's policy for Racially Isolated Minority Schools (RIMS). The 
method has three steps: (1) establishing e RIMS-specif ic District 
level data base containing data on achievement, student demographics, 
school demographics, and classroom practices for implementing 
instructional programs in RIM schools; (2) developing an analysis 
design for RIMS-specif ic data which identifies skills students do 
have (rather than "missing" skills) and has significant implications 
for adjusting RIMS schooling practices; and (3) using analyses which 
identify successful instructional practices to implement a two- to 
three-year instructional plan for moving achievement along in RIM 
schools. This methodology provides an affirmative complement to 
traditional measurement and evaluation methodology used to analyze 
conventional achievement test data. (BS) 



*********************************************************************** 

* Reproductions supplied by EDRS are the best that can be made * 

* from the original document. * 
*********************************************************************** 



ERIC 



ERIC 



\h\\ dtuument h.is been distributed to j limited audience for a limited purpose It is not published (\)pics may be 
fn.ide only v^jfh the written permission of SWRL J-Jucalional Re^arch and Development. 4665 Lampson Avenue. Los 
M.jnutos, California The « upon whuh (his document is based was performed pursuant to Contract 

Q \\ i\.\H).\.{){\tA with the National Institute ol Education SWRL reports do not necessarily reflect the opinions or 

pi>hcies ot the 'iponsors of SWRL R&D. r\ 



/ 




SWRL EDUCATIONAL RESEARCH AND DEVELOPMENT 

TECHNICAL REPORT 70 
April 6, 1981 



METHODOLOGY FOR ANALYSIS OF lAI DISTRICT LEVEL DATA BASES 

•J 

Patricia Mi lazzo, Aaron Buchanan, and Richard E. Schutz 



ABSTRACT 

Data bases yielded by Instructional Accomplishment information Systems 
provide the opportunity for new and powerful studies relevant to policy 
issues of concern in education at a local and/or national level. The 
methodology for such studies is discussed. Procedures are depicted using 
Racially Isolated Minority Schools in Large Urban Districts as an 
i 1 lustrat ion . 
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METHODOLOGY FOR ANALYSIS OF lAI DISTRICT LEVEL DATA BASES 
Patricia Mllazzo, Aaron Buchanan, Richard Schutz 



Instructional Accomplishment Information (lAl) Systems represent a 
new approach to Instructional program measurement and evaluation. On 
superficial Inspection, lAI Instrumentation' can iook much like 
counterparts associated with achievement testing, but there are funda- 
mental differences In lAI systems and conventional achievement testing 
systtns In how the instrumentation Is derived and how the results are 
used.^ Just as lAI student and teacher materials can look like routine 
tests, an lAI data base at the district level can look like routine 
compilations of "test" scores. Conventional measurement and evaluation 
methodology can, of course, be applied to any lAI data base, but this 
practice loses the prime value of the data. Extracting the unique 
information residing in an lAI district level data base requires appro- 
priate methodology. This methodological aspect of lAI is the subject 
of the present paper. 

Like other aspects of lAI systems, the methodology for analyzing 
lAI data bases is simple compared to the complicated statistical and 
inferential apparatus that is currently associated with the analysis 
and ■ ,terpretation of achievement test results. At the district level 
of aggregation an lAI data base has rich potential for analyses that 
can illuminate and guide school Improvement Initiatives In the district 
and at the same time yield better general understanding of schooling 
and Its accomplishments. 

A district level lAI data base is suited ideally for two general 
categories of analysis: school Inq subject studies and school inq pel icy 
studies. Schooling subject studies query the data base from the 
orientation of a given academic subject (e.g., reading, mathematics, 
composition, science, etc.). Currently, the methodology for such 
studies Is most conveniently depicted by reference to summary reports 
of several studies that have been conducted in specific academic 
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subject areas. For example, see Buchanan and Mllazzo, 1978a and 1978b; 
Flege-Kollmann, 1978 and 1980; Buchanan and Perkins, 1980; Cronnel 1 and 
Humes, 1980; Lawlor et al., 1981. For this reason, this paper will not 
deal further with the description of the methodology for schooling 
subject studies. 

Schooling policy studies query the data base from the orientation 
of a given administrative policy. Administrative policies can take 
several forms. Currently the "hot" policies In schooling relate to the 
Intersect of professional and demographic characteristics that are leg- 
islatively or judicially mandated {e.g., educationally disadvantaged, 
handicapped, and bilingual). The two general categories of subject 
studies and policy studies, then, could also be distinguished fundamen- 
tally as dealing with matters of quality and equality In education. 

Schooling policy studies Involvlnrj lAI data bases first became 
feasible with the 1980-81 data base for the Los Angeles School Dis- 
trict's Grade-by-Grade Advancement Program. Some sense of the 
methodology can be gained by reading the one analysis that has been 
completed (Mllazzo, 198l). However, the methodology is more conve- 
niently depicted by sketching the method for an analysis yet to be 
completed. The policy at Issue in the illustration pertains to a 
matter of local and national concern: racially Isolated minority 
schools. It should be evident through the example that the methodology 
is readily pertinent to other matters of current policy concern in 
education. 

Illustration of Schooling Policy Study Method 

The administrative policy relevant to the study Is the LAUSD 
policy for Racially Isolated Minority Schools (RIMS). The policy is 
important to Los Angeles Unified School District because RIMS is a key 
element in the District's program of school excellence. Information 
regarding Instructional accomplishments in RIM schools is particularly 
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timely now, given the predictably critical "evaluations" the District 
will receive fron several outside agencies relative to their integra- 
tion program. If these evaluations progress the way integration 
evaluations almost always progress, we will read still another spate of 
comparison data showing us that there is a "significant relationship 
between ethnicity and performance scores; that the majority of students 
who demonstrate serious achievement deficiencies are In RIM schools; 
that the majority of RIMS students have not mastered basic skills, 
given a pre-established standard of acceptable performance (certainly 
more than 60? of RIMS students at grades 4-6 will be identified as 
•failing' if overall achievement levels of 70^ and 801 are used as the 
standard); that many Hispanic children and just about all black 
children in the District are in RIM schools," and so on. 

None of the findings generated by the conventional evaluation 
studies on instruction is news; the well-known situation Is simply 
restated in each new study. But the publicity that surrounds these 
findings is news, and Its consequences can only be detrimental. That 
is, the typical evaluation of instruction In RIM schools is grounded in 
the tacit assumption that learning is an "all or nothing" event, and 
the Inevitable follow-up to this kind of evaluation is yet another pro- 
gram of RIMS remedial instruction concentrated almost exclusively on a 
collage of "missing" skills where instruction has already failed. The 
net effect is to truncate both instruction and achievement for minority 
students. Alternative analyses, however, are possible, grounded in the 
assumption that important learning from past instruction can always be 
demonstrated and that new Instruction can be developed around new pri- 
orities for expanding proficiencies that students already have. The 
study sketched below gives alternative data analyses and implications 
for instructional planning. 

The analysis is also pertinent to a better understanding of 
schooling practices and effects generally. If nothing else has been 
learned from studies over the past two decades of the cont«nporary 



American social syst«yn and of public schools as part of that social 
system, It is that the elementary grades are the last good chance most 
children 'in RIMS environments typically have. Students who leave the 
elementary grades without expanding a specific, identifiable set of 
skills (which most RIMS students can demonstrate) seldom n.ake it In 
high school, or later In conventional life. Many of these children 
drop out of high school, and still others who do graduate have not 
expanded even the most rudimentary skills ostensibly learned in school. 
It is difficult to see how the si tuat i on can do anything but get more 
urgent. What Is needed is Information to justify specific operational 
efforts to dramatically affect schooling and learning in these schools. 

The method can be described in three steps: 

• The first is to establish a RIMS-specif ic data base at the 
District level containing data on achievement, student demo- 
graphics, school demographics, and classroom practices for 
implementing instructional programs In RIM schools. 

• The second activity provides a design for analyzing the 
RIMS-specifIc data and lodging implications for adjustments in 
RIMS schooling practices. 

• The third set of activities lets us use these analyses to 
implement a two-to-three year Instructional plan for moving 
achievement along in RIM schools. Here we operational Ize 
strategy for building on skills that RIMS students do have in 
place and for expanding the range of those skills, Instead of 
looking for skills that are "missing" and then somehow trying 
to f i 1 1 the gaps; 

These steps are elaborated In the description that follows.^ 

Step 1: Establishment of a District Level RIMS-Specif ic Data Base 

The key component of this data base is the instructional 
accomplishment Information derived from the Survey of Essential Skills 
(SES) used In LAUSD. To generate this component, the S% random sample 
that is already a part of the SES operating system can readily be used 
by simply duplicating the RIMS data in another file. The 5* sample is 
a representative District sample and hence a representative RIMS sample 
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(about 501 of the 5* sample, or some ^500 RIMS students). A sample 
this size provides all the power and efficiency that is needed. More- 
over, it provides a quick, inexpensive way for getting longitudinal 
data because there are already random samples for 1979 and 1980, and 
another sample for 198i will become available this Spring. 

This will generate two major elements of the District level data 
base: 1) grade-by-grade achievement scores for RIMS students on the 
SES and 2) grade-by-grade student demographics for RIMS. Analyses can 
go a long way with just those data. But three more elements are needed 
for comprehensive analyses: 

• One, a file of school level demographic data on each RIM school 
is needed. The records should be left open-ended, because no 
doubt other variables will be added as time goes on. It is 
evident now that there is pertinence in data on variables such 
as: 

- neighborhood/socio-economic status data 

- student body size/classroom size 

- categorical classifications 

- release rate (frequency for early dismissal during the 
school day for teacher conferences, special meetings, 
special events, etc.) 

- percent of days dedicated to formal, school-level 
assessment 

- rate of teacher absenteeism 

- physical characteristics of school facility 

- discipline indicators (e.g., referral rates) 

Most of this information, and a lot more, is already a part 
of District record. The major concern is to be careful to 
keep this file manageable and meaningful, and to avoid 
including data on variables just because they can be 
collected. * 

• Two, a summary of school achievement scores is needed, for 
every grade and for every SES skills category and every 
Item. These data are already available on school summary 
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tapes compiled annually to generate Individual school 
reports for the SES September reporting period. Data from 
RiH schools can be sorted Into a separate file. S nee the 
Nummary school data bases contain aggregated school data, 
this procedure will provide aggregated scores for aH RIMS 
students, grade-by.grade, on the Individual SES Items and on 
clusters of Items as well. 

The last element of the data base has to do with classroom 
practices data. These data will provide the most edifying 
analyses, and probably the most significant Insight into the 
adjustments that are needed In RIMS Instruction. Just the 
minimal amount of exploration done to date with the help of 
the LAUSD Research and Evaluation division has revealed 
several classroom practices like the following ones that 
were different In suburban schools where SES scores were 
higher than RIMS; In fact, they were different for the few 
RIM schools where SES scores were higher than other RIM 
school s: 

- start-up date for teaching each subject 

- ending date for completing Instruction on each subject 

- number of days each wfeek that instruction Is provided 
In a specific subject,; area 

- textbook program used In a specific subject area 

- average length of a classroom lesson In a specific 
subject area 

- percentage of textbook lessons completed In a school 
year 

- percent of class time dedicated to testing 

- length of use of a particular textbook series In a 
specific subject area 

- length of use of a particular teaching strategy for a 
specific subject area 

- percentage of substitute teacher days 

- percentage of classroom time controlled by specialists 
in each subject area 

- percentage of school day spent disciplining students 
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- percentage of school day spent directing aides 

- percentage of aide-directed classroom lessons 

Step 2: Design for Analyzing RIMS-Specl f Ic Data and T heir Implications 
for Instruction 

In setting out to look at RIMS accomplishments, a variety of 
standard data tables can be routinely generated. For example, computer 
processing can readily provide grade-by-grade achievement scores broken 
out by language and ethnicity, by the various categorical classifica- 
tions, by teacher absenteeism rates, by the several typical school 
socio-economic status categories, by textbook programs, number of days 
of instruction each week In a subject area, amount of lessons completed 
during the school year— all of t'he variables Identified In the earlier 
] I St i ngs. 

Computer analysis can also Interrelate key elements from each type 
of data basej school demographics, student achievement and demograph- 
ics, and classroom practices. This type of activity Is more difficult 
to accomplish than the previous one. But these data analyses will be 
the key to a RIMS Improvement plan. What this type of data analysis 
will not do Is Interrelate every variable to every other variable In 
every likely extrapolation, and generate a mound of output. It Is 
unnecessary to engage In that sort of exploratory data analysis, 
because the keys to the kingdom are already In hand In the form of 
achievement scores. The first and cardinal rule In querying RIMS data 
bases Is to look at the aggregate of RIMS students who have accomplish- 
ment scores that are at or above the district aver age; or. even better, 
at or above the average for non-RIMS students . The commitment is to 
isolating RIMS accomplishments, and not Just because It is beneficial 
to point to them— although that would be a refreshing change for RIMS. 
The Intention is to Isolate RIMS accomplishments so that they^can be 
traced back to practices that schools have some control over. This 
type of analysis looks at schooling practices relative to RIMS students 
who demonstrate accomplishments, because the search is for what works 
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in RIMS, to find It and hang on to It, and to generalize It to other 
RIMS. Wbat is finally sought Is an annual profile of successful 
schooling practices in RIMS. Of course, the analysis will look annu- 
ally at overall RIMS accomplishment data, and that Is discussed below. 
But every time one leads with such summaries, the first question faced 
is "But how do RIMS compare with white, suburban schools?" This level 
of gross comparison shows only that the scores for the overall group of 
white students Is higher than for thi overall group of black students. 
But that Is when people quit listening, thinking they know the rest of 
the story, without hearing It— nothing has been adcompl i shed. 

An example of the method Is useful here. In 1980, the average 
mathematics score for low absentee grade 6 white students was 71%. The 
average mathematics score for a similar group of black students was 
60*. This m score Is seriously depressed by the fact that many of 
the lowest scores In the District are In black RIMS. Hawever, some 
black RIMS students are pulling the RIMS average up. We queried the S% 
sample data base and found that about 20% of the RIMS group was at or 
above the 73% level. Such data can be obtained easily from the 51 
sample. We can then relate this group of hlgh-accompl Ishments RIMS 
students to their schools and teachers, because each student record in 
the 5% sample Is led by unique school Identification and teacher name. 
When a classroom practices file Is completed, It will be possible to 
accumulate a profile of practices that typically associate with high- 
achievement RIMS. In addition, a few teachers can be further consulted 
via a one page sheet of information which can be keyed into a computer 
file. The data collection would probably Involve Just a few teachers 
and could be completed In a week or two. By leading this way, atten- 
tion moves away from ethnicity (where It doesn't belong) to Instruction 
and practices (where It does belong). Moreover, the Information can 
effectively respond to Individuals and Interest groups who want to con- 
tend that any kind of exploratory data analysis is intended only to 
explain away the "fact" that RIMS don't teach their students anything. 
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This approach will provide a major source of descriptive 
information for the District's prospective planning for RIMS Instruc- 
tlon. However, It Is Important to also recognize that the District 
will want to make presentations to the Board and the public about the 
general state of affairs within the RIMS on at least an annual basis. 
For this purpose it is reasonable to 11 lumlnate patterns of accomplish- 
ments on the various skl41s categories at each grade level. For 
example, the two graphs on the next page show mathematics performance 
for RIMS students at grade 3 (graph A) and at grade 6 (graph B). In 
each grade, the graphs show a break out of RIMS averages on all of the 
major skill categories which were assessed (the numbered curves), and 
by high, medium, or low absenteeism. These several skill area averages 
are displayed around the overall District average for the grade level 
(the heavier line which Is unnumbered). This District average is for 
all students who took the SES. On Inspecting Graphs A and B, it can 
readily be seen that RIMS demonstrate Instructional strengths at each 
grade level. At grade 3, most of the seven broad skill areas which 
were assessed hover around the overall District average, and it is two 
skill areas which seriously depress the average for black students at 
grade 3. Already, the instructional problem is showing some signs of 
being controllable. What's more, it is clear that students who have 
iow absenteeism rates are really quite close to the District average, 
except on the two skill areas, numeration and problem solving. There 
are a few pieces of information that don't show on this chart, but they 
help our understanding. First, numeration and problem solving were 
difficult for aJJ. students. Of the seven skill categories in grade 3» 
problem solving had the lowest score and numeration had the next Icwest 
score--for everybody. Second, when we look at the specific skill 
descriptors for numeration and problem solving, we see some explanation 
for low scores. Problem solving, for example, Includes three Items on 
selecting a number sentence to describe a real life problem, or vice 
versa, and those Items account for the lowest scores— for everybody. 
When one looks to find this skill reflected In lessons in the textbooks 
or other instructional materials that are in wide use In the District, 
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BREAKDOWN OF SKILLS CATEGORIES SCORES FOR BLACK STUDENTS 



MATHEMATICS: GRADE 3 




Graph A 

Skills cattgori^ Indtx: 
1) numaratiort 

2) 

3) xl^ 

4) fractions 

5) mtasursmsnt 

6) rtlations/functions 

7) probism toivlng 

UNNUMBERED: district averags for all 
34,789 students who took the grade 3 
SES (all ethnic groups) 
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MATHEMATICS: GRADE 6 




Graph B 

Skills categories index: 
1) 

2) x/^ 

3) fractional numbers 

4) decimals 

5) geometry 

6) measurement 

7) reiatlons/functions/statistics 

8) problem solving 

UNNUMBERED: district average for all 
37,653 students who took the grade 6 
SES (all ethnic groups) 
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it can hardly be found. ' This means that as things stand, teachers may 
have to locate or develop many of their own lessons In order to teach 
this type of problem solvlig, an unlikely event In most classrooms and 
highly unlikely In RIMS classrooms where teachers' time Is seen to be 
at 'a premium even more than for most schools. Again, when viewed 
through this sort of clearly focused lens, the achievement picture for 
third grade black students In LAUSO Is very encouraging. 

Now look at the grade 6 graph (graph B). Two of the skill areas 
assessed In grade 6 (+ and x and t), are well above the District 
average. When we look at the specific skill descriptors for the con- 
tent of these skin categories, we see skills that are reflected In 
several lessons within grade levels, and are strengthened over two or 
three grade levels. On the other hand, when we look at the skills 
represented In geometry, measurement, and relations/functions/ 
statistics, we have a harder time finding the skills reflected In dis- 
crete lessons in Instructional materials. There Is usually something 
taught, but the opportunity to teach and learn Is seriously restricted. 
Again, this situation has serious Implications for Instruction in 
general; but the Implications are much more serious for RIMS, where 
teachers' time is in very short supply. 

One last point, before we go on. Look at fractions in the grade 3 
chart and then in the grade 6 chart. At grade 3, black children do 
considerably bette. on fractions than the District's overall mathemat- 
ics score, but at grade 6 they do a lot worse. That has nothing to do 
with "being black" or with "declining" growth In ability to do frac- 
tions. The pattern is typical for all of thu District's ethnic groups, 
and the explanation seems to reside in Instruction. Fractions In grade 
6 are nothing like fractions In grade 3. The matter Is not one of 
declining achievement, it Is an Instructional matter that reflects two 
different situations with a single label. In fact, about the«only 
skill categories that maintain their rank across grade levels are the 
computation skills, which are high, and problem solving, measurement, 

is 
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and relatlons/functlons/statlstlcs, which are low. Given the previous 
discussion, one could have predicted that ranking order. The reading 
story and composition story conclude the same way. 

For the large part, all of the demographic examples get this same 
l<ind of discussion, so that the Implications for Instruction are com- 
mon, regardless o* the demographic features that are looked at. On the 
other hand, the Implications for RIMS, when one looks at eff:xtlve 
classroom practices and the general strain on teacher time, are nicely 
specific, and these can be outlined In Otep 3. 

There Is one demographic factor that appears to be dramatically 
different; language-especlally for students who are identified as NES 
at the beginning of the school year, but even for LES students. How- 
ever, the language factor, which looks critical for the reading subject 
area (not surprisingly), really Is not a serious problem for mathemat- 
ICS. Look at graphs C and D on page 13. They are adml ttedl y full , but 
they tell the story. Graph C shows the average SES scores at each 
grade level for NES (bottom curve), LES (middle curve), and for the 
District overall (top curve). The percentages on the LES and NES 
curves are a bit confusing without further explanation. Look at the 
grade 1 points for NES and LES. Five to 7 percent of all students who 
took the grade 1 survey were NES; 10 to 111 of all students who took 
the grade 1 survey were LES. Since the District demographics showed 
approximately 71 NES and 111 LES overall, we can assume that most of 
the NES/LES students took the grade 1 reading survey. That is notable 
because the NES teachers had a choice, except In grade 6, where every 
student In the District regardless of language fluency took the SES. 
Across all the grades, pretty much all the LES students took the S^S 
surveys. However, in grades 3, ^, and 5, NES teachers tended to exer- 
cise their option to not assess their students. Only 1 or 2% of the 
students at these grade levels were opted into the surveys. That is 
notable, too, because we are probably looking at the "best" NES stu- 
dents in grades 3, ^, and 5; so the reading averages at those grade 
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READING 
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levels are probably higher than they would be If we had all of the NES 
students, taking the SES. In fact, they would probably look like the 
grade 6 average, which Is almost guess level. What all of this Infor- 
mation means, bluntly, Is that English language reading proficiency for 
NES students at grades 3-6 is not being attained. In part, this Is 
predetermined by the designation NES. Graph 0, however. Indicates that 
the determinants lie in pedagogy rather than elsewhere; mathematics 
Instruction doesn't suffer too much from lack of English language 
fluency. 

Absenteeism was generally an important consideration, no matter 
what the demographic characteristics. Students who were absent more 
than 20 days tended to have averages that were approximately 10^ lower 
than averages for students who were absent less than 10 days. "101" is 
an important number because it turns up so much. "10%" Is also about 
the difference in proficiency between one year and the next on skills 
that are taught and retaught. About 101 of the students are picked up 
on skills which are retaught a second year, so we are likely looking at 
a grade-by-grade effect. 

At grades 3-6, where there were the greatest differences between 
ethnic groups In proficiency patterns, there were little or no differ- 
ences between ethnic groups in terms of absenteeism patterns. Hispanic 
children were absent more than other ethnic groups, but not much. The 
greatest differences in attendance patterns between ethnic groups came 
in grades 1 and 2 where black children and Hispanic children tended to 
be absent considerably more than other ethnic groups. Even though 
grades 1 and 2 are where we see the least differences between ethnic 
groups in terms of proficiency patterns, this decreased attendance by 
black children and Hispanic children at grades 1 and 2 may certainly 
have a cumulative effect on achievement In grades 3-6. Because profi- 
ciency at grades 1 and 2 is generally high, any effects of higher 
absenteeism probably would not show until later grades anyway. 
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These examples give an Image of the story on the demographic 
analysis. Our exp^loratlon always leads to powerful and plausible 
explanations In terms of the expectations In the LAUSD skills 
continuum, their congruence or Incongruence with actual lessons in 
Instructional materials, and the availability of teacher time to 
produce lessons where they don't exist. 

Step 3: Initiatives for Instructional Improvement 

The Initiatives for Improvement of Instruction build directly on 
the analytical capability for Identifying successes In RIM schools 
wherever they occur and for tying these successes as much as possible , 
to antecedents over which the school is likely to be able to exert some 
control. Many of these controllable antecedents to Instruction have 
already been alluded to. Other, obvious ones Involve the use of 
instructional time and space, characteristics of teaching staff, school 
attendance, amount of support from parents, special physical or 
emotional/behavioral problems, and so on. The strategy is to locate 
successful practices and to build upon them, and It requires two 
distinct kinds of Initiatives: 

• Pre- Instruction initiatives . These Initiatives Involve 
antecedents to success that are beyond the direct scope of 
Instruction but are nevertheless ones over which a school may 
be able to exert some control. If students are absent or late 
upwards of ten or more days during the year, then ostensibly 
the school can mount some initiative to improve attendance, or. 
In case of Illness, to send special assignments home. Where 
the level of parent Involvement with student work In school Is 
low, Initiatives can be undertaken to provide parents with 
better Information about student needs and accomplishments 
Including samples of the kinds of performances that students 
can and cannot manage. In the long run, obtaining parent 
support directly In Identifying and building on student accom- 
plishments may well be more productive than Informing them 
about general problems with attitude, study habits, and 
classroom behavior. 

e 

• Instructional Initiatives . The most Important aspect of 
Initiatives for direct Improvement of Instruction Is specific 
staff development. For RIM schools, it has to be assumed that 
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the materials used now are about as good as the current state 
of the art In textbooks. Anyhow, the way teachers use materi- 
als (or make them work) is probably most Important. There Is 
one exception. Skills that are expected to be delivered In 
schools must have strong representation In either print mater - 
als or In detailed guides for teachers to use In creating their 
own activities. Especially In RIM schools, teachers can t be 
expected to deal with large numbers of students for whom 
success Is likely to be marginal at best and still be able to 
develop Instructional resources on the fly. The rule of thumb 
that should operate In these schools Is that places where a 
reasonable amount of materials don't exist, say at least four 
or five Integrated lessons, district expectations In reading, 
mathematics, and language arts don't apply. 

The most Important element of actual Instruction Is to build on 
successes that Individual students already have. The assertion 
may seem obvious, but It's a fairly wide departure from 
prevailing approaches In the U.S. to elementary schooling 
Instruction, especially In schools where students are typ cally 
quite far behind the pace for regular grade-by-grade ski. I 
development. The dominant model Is for remediation. Teachers 
have a pre-established network of skills that students are 
supposed to hav3 and proceed with Individual students to fll 
In missing pieces. Ideally, In the remedial model, instruction 
is heavily Individualized with many different students working 
on quite different skills. Building on successes or accom- 
plishments Is not the same. Here teachers look for medium to 
high scores rather than low ones and go to work there. In this 
case, students learn to apply skills before they begin to 
develop new ones. The process is more like construction than 
remediation, since It Implies that the most Immediate goal Is 
to increase and Improve range of operation or application or, 
more specifically, conditions under which performance can be 
expected to be successful. Two assumptions are Involved. 
First, students who are one to two years behind the pace of 
regular grade-by-grade expectations cannot be expected to catch 
up "naturally" or by routine remediation. Second, there are 
indeed children who may not progress Instruct lonal ly despite 
the schools' best effort. Often they are students who are not 
In school and not likely to come. This does not include stu- 
dents who are in school but are simply hard to teach. The idea 
is to reduce the set of students that cannot be reached as much 
as possible and to take responsibility and direct resources 
where schooling has some chance of having an Impact. 

The implications for staff development are far reaching. First 
of all, teachers need help In getting past guilt for those 
expectations that cannot possibly be met, at least not directly 
or within a short range of time. They need help In beginning 
to Identify potential for Immediate growth In students. They 
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need help In reorganizing skill development so that success can 
be built In early and often, and sets of activities built to 
closure In no more than five to ten lessons. 

Second, RIM schools peed assistance In plans for dealing in 
simple, realistic ways with students who are clearly behind the 
pace of regular Instruction and to be able to show how Instruc- 
tional Initiatives have been Implemented In two ways. They 
need to be able to obtain regular and explainable Information 
on student completion of work. The purpose Is to provide 
administrators In RlM schools with better data on how instruc- 
tion Is being delivered, not as a device to make teachers njore j, 
accountable, but to give credit for teaching effort In cases 
where a great deal of progress still may not have been made. 
Uslna the same technology, they also need to Identify student 
success on a regular and explainable basis and to get this 
Information Into a more accessible form for personnel who are 
responsible for research and analysis. Both kinds of data 
collection can. If desired, be Integrated nicely with end-of- 
year data obtained from administration of SES. 



Final Note 

The methodology depicted in the previous pages provides an 
awlrmatlve complement to traditional measurement and evaluation meth- 
odology which has been used to analyze conventional achievement test 
data. This methodology Is consistent with a new awareness In the fiel 
(Cronbach and Associates, 1980) that Instructional program evaluation 
and school Improvement can all too easily get side-tracked by the 
asceptic compulsiveness which often characterizes "analysis of vari- 
ance" and "true experiment." The methodology sketched here appears a 
useful mechanicsm for accomplishing the technical and professional 
Intentions of the "new view" of program evaluation. 
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NOTES 



Examples Include the Los Angeles Unified School DIstr cfs %}pj^ 
of Essen tial Skills , 1979-1982 editions; Sacramento City Grade 5 
pT^ fiTiTncy Survey ; SWRL Proficiency Verification Sy stems 
Inventories ; the planned Washington D C Assessments, 1982 edition. 

Definitive descriptions can be found Int Designing an 
in.t rnrtlQn-Referenced System for Large- Sr.^le Evaluation of School 
AchievementT Aaron D. Buchanan and Patricia A. Mliazzo, P^-^ented 
at the Thir d Annual Conference of the California Society of Educa- 
tional Program Auditors and Evaluators, Los Angeles, May •Z-«3, 
1977, ^nH P\/S Coordinators Manual , SWRL Educational Research and 
Development, 197$. Concise descriptions of the differences between 
the various achievement Instrumentation are found in: What Makes , 
Ach i ev ement Tests Tick? invest igation of Alternative Instrumen- 
tation for Effective Program Evaluation , Ralph Hanson, Richard 
Schutz, and Jerry Bailey, a SWRL document, 1980; and Equating 
Instructi onal Accomplishment Invent ories and Standardized 
Achi evement Tests , Patricia Milazzo and Aaron 0. Buchanan, a SWRL 
document, 1980. " Other more recent descriptions, intended largely 
for the lay public Include: Scope of Work for Meeting Competency 
Requ I rem ents of School Districts Grade-by- Grade Assessment and 
AHvl^^ nrementt Desion. Dev^opment. and Verification of Implementa- 
tion Resour ces and Procedures , a SWRL working paper, 190O; 
Pr oficiency Ver If Icatlon S7?tems, 1981 Brief , a SWRL document, 
"iq8l: Design Parameters for Development of an Information System 
from a~Skll Is Framework , Aaron Buchanan and Patricia Mllazzo, a 
SWRL document, I960. 

The description is for purposes of exposition only. In this sense 
it is strictly hypothetical. There is no commitment, express 
or implied, of any agency, institution, organization, or individual 
to the description. 
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